Communication Event

ABSTRACT

In a communication event between a first user and one or more second users via a communication network. A plurality of video streams is received via the network at a computer connected to the network. Each of the streams carries a respective moving image of at least one respective user. The computer causes the respective moving image of a first of the video streams to be displayed at a user device of the first user for a first time interval. The computer detects, in the respective moving image of a second of the video streams that is not displayed at the user device in the first time interval, a change in the visual content thereof. In response to detecting said change in the visual content, the computer causes at least the second video stream to be displayed at the user device for a second time interval.

RELATED APPLICATIONS

This application is a continuation of and claims priority to U.S.application Ser. No. 15/624,297, entitled “Communication Event,” filedJun. 15, 2017, which is a continuation of and claims priority to U.S.application Ser. No. 15/048,800, entitled “Communication Event,” filedFeb. 19, 2016, which are incorporated herein in their entirety.

BACKGROUND

Voice over internet protocol (“VoIP”) communication systems allow theuser of a device to make calls across a communication network. To useVoIP, the user must install and execute client software on their device.The client software provides the VoIP connections as well as otherfunctions such as registration and authentication. Advantageously, inaddition to voice and video communication, the client may also providevideo calling and instant messaging (“IM”). With video calling, thecallers are able to view video images (i.e. moving images) of the otherparty in addition to voice information. This enables a much more naturalcommunication between the parties, as facial expressions are alsocommunicated, thereby making video calls more comparable to aface-to-face conversation.

A video call comprising multiple users may be referred to as a “videoconference”. In a conventional video conference, each participant (i.e.user) is able to view the video images of one or more of the otherparticipants (users) in the video conference. For example, as a defaultsetting, each user may be presented with the video images of all of theother users in the video conference. These may displayed, for example,using a grid, with each video image occupying a different location onthe grid. Alternatively, each user may be presented with one or morevideo images corresponding to users that have been detected as speakingusers. That is, the detection of audio from a speaker may determinewhich of the video images of the other users are selected for display ata particular user's user terminal. Typically, in a video conference, oneuser speaks at a time, and so this may result in a single video image ofthat user being displayed to each of the non-speaking users.

SUMMARY

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter.

Various aspects of the present subject matter relate to a communicationevent between a first user and one or more second users via acommunication network. A plurality of video streams is received via thenetwork at a computer connected to the network. Each of the streamscarries a respective moving image of at least one respective user. Thecomputer causes the respective moving image of a first of the videostreams to be displayed at a user device of the first user for a firsttime interval. The computer detects, in the respective moving image of asecond of the video streams that is not displayed at the user device inthe first time interval, a change in the visual content thereof. Inresponse to detecting said change in the visual content, the computercauses at least the second video stream to be displayed at the userdevice for a second time interval.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the present subject matter, and to showhow embodiments of the same may be carried into effect, reference ismade to the following figures in which:

FIG. 1 shows a schematic block diagram of a communication system;

FIG. 2 shows a schematic block diagram of a user device;

FIG. 3 shows a functional block diagram of a server;

FIG. 4A shows a schematic illustration of a computer implementeddatabase that may be used to determine a priority associated with avideo stream, based on an identified reaction in the moving image of thevideo stream.

FIG. 4B shows a schematic illustration of a computer implementeddatabase that may be used to determine a priority associated with avideo stream, based on a detected change in the number of users in themoving image of the video stream.

FIG. 5A shows a flow chart for a method of selecting video streams fordisplaying at a user device during a call, based on the identificationof an expected movement in the moving images of the video streams.

FIG. 5B shows a flow chart for a method of selecting video streams fordisplaying at a user device during a call, based on a detected change inthe number of users in the moving images of the video streams.

FIGS. 6A to 8C show various illustrations of a graphical user interfaceof a client, at different stages during a video call between a group ofthree or more users;

FIG. 9 schematically illustrates selectable predetermined layouts.

DETAILED DESCRIPTION

Overview

In a video conference conducted via a communication network, it may notalways be desirable to display all of the video images of the otherusers in the videoconference, to a particular user. This may be thecase, for example, where only a few of the users are active—i.e. doingsomething that may be of interest to one or more of the other users inthe videoconference and/or if there is a very large number of users onthe call. For example, only one or two of the users may be speakingusers. It may be desirable to prevent the video images associated withthe remaining users, i.e. the inactive users, from being displayed at auser's terminal. This ensures that the user terminal does not allocatedisplay resources to video data that does not add to the user'sexperience of the video conference. This is particularly, though notexclusively applicable, to mobile, tablet or certain laptop deviceswithin limited available display areas. It may also in some cases ensurethat network-bandwidth is not allocated to transmitting the videostreams associated with the inactive users to other user terminalsunnecessarily, as discussed in further detail below.

This is referred to herein as “follow the action storytelling”, andguides the consuming participants with the group activity and groupresponse as the communication event proceeds. Currently, with activespeaking video conferences, the consuming participants may need tomonitor multiple video feeds at once to determine where non-verbalactivity is occurring. Alternatively, if they are only viewing activespeaker they may not be aware of non-verbal changes in the groupactivity due to the absence of any suitable visual representation beingpresented to them.

It may not always be desirable to only display video images of usersthat are identified as speaking (i.e. displaying video images basedexclusively on the detection of verbal events). For example, a user maybe interested in a non-verbal event associated with one or more of theother users. This may include an event such as a detected change in thenumber of users that are present in the video image received from aparticular user terminal. It may be desirable to display this kind ofevent to a user, in addition to, or in place of, the video imagesassociated with verbal-events. In doing so, a user may be able to viewthe activity of the other users in the video conference (be it verbal ornon-verbal) in a story-like manner. It may be desirable to displaynon-verbal events for a limited time interval; for example, to ensurethat a user's focus is not taken away from a speaking user for too long.

Furthermore, given that a non-verbal event can be conveyed visually,i.e. without the need to play out any associated audio data, it may notbe desirable to replace both the audio and video data associated with aspeaking user, with the audio and video data associated with the videoimage in which the non-verbal event was detected. For example, it may bedesirable to ensure that a speaking user's speech is still played out ata user's user terminal, even if the video image that is being displayedat that user's user terminal, does not correspond to the video image ofthe speaking user (e.g. if the video image corresponds to the videoimage in which a change in the number of users was detected). Treatingthe audio and video data in this way ensures story continuity of thegroup experience—i.e. that a user's focus is brought to the relevantaudio and video data, at the right time in the video conference.

The present disclosure addresses these issues by providing acommunication system that causes one or more video images of a videoconference to be displayed at a user terminal in a virtual “StoriedExperience View”. The virtual “Storied Experience View” harnesses thepower of video and storytelling to transform a meeting experience (i.e.video conference) beyond active speaking via a more engaging andlife-like meet up experience.

The Storied Experience View may comprise a single streaming video gridor a multi-streaming video grid where multiple videos and or audiochannels play at one time. By displaying video images in the StoriedExperience View, users are able to consume the most engaging and tellingstory of group activity, i.e. without having to monitor all of the videoimages of all of the other users in the video conference, in order todetermine where non-verbal user activity is occurring. In the StoriedExperience View, video may be displayed or sequenced due to the currentaction at hand using a combination of multiple video grids and singlevideo grids depending on the size of the group activity and the numberof sensors capturing the event.

In the present disclosure, one computer receives all of the videostreams from each of the respective users, via the network so that anintelligent decision about which to display can be made taking intoaccount all of their visual content. The computer has visibility of allof the candidate streams and is able to limit the number of these thatare selected for display at a particular user terminal, taking intoaccount

-   non-verbal events i.e. changes in the visual content of the moving    images carried by the streams. Because the computer receives all of    the video streams via the network, it is best placed to make    intelligent decisions about which video streams to select. Limiting    the number of video streams in this way is useful, where, for    example, a user terminal has a limited display area. In such a case,    it may not be meaningful to display all of the video images of the    other users in the video conference, at that user terminal    (particularly if the video conference has a large number of    participants). The computer receiving the streams is able to work    within the confines of the limited display area whilst maximizing    the information that is delivered to the consuming user.

This is particularly, though not exclusively, the case where at leasttwo of the video streams are received from different clients running ondifferent client devices as each individual client is not necessarilyaware of the visual content of the other client's video stream(s).

In the described embodiments, the computer is embodied in a centralserver. This allows bandwidth to be saved, as the only the stream(s)selected for displaying to any given client need to be transmitted tothat client from the server. In this way, the server is able to usebandwidth efficiently, whilst maximising the amount of useful and/orengaging information that is conveyed to a consuming user.

In embodiments of the present disclosure, detection of a change in thenumber of people at a specific remote location video stream is, forexample, used to trigger a wide framing story sequence priority in theconsuming video story experience, thereby providing a live, visualawareness of the enter/leave activity of the people at the remotelocation to the consuming attendees as it changes by displaying apreviously non-displayed video image having a wide view, e.g. of aconference room in which multiple users are located, when one or more ofthose users leaves and/or when one or more new user's join them. Thatis, in addition to displaying previously non-displayed images inresponse to detecting enter/leave events in them, images may, in someembodiments of the present subject matter, be selectively cropped beforedisplaying them in dependence on such events. The detection is performedon the uncropped image, to ensure that changes in the visual contentthat may have been cropped out are still detected. For example, when anenter/leave event is detected in a video image that is not beingdisplayed, that may cause the image to be displayed in a wide view (i.e.with no or minimal cropping) to ensure that the event is visible. Thisalso applies to an enter/leave event detected in a video that iscurrently being displayed. For example, if the active consumed video iscropped to an active speaker in the total video view (medium or close upof head and shoulders) a change in the number of users via face, audio,movement, may be detected in the uncropped version of that video image.In response, the video is un-cropped to the widest viewable view sousers can see the change in the number of people in the room (plus orminus).

That is, when a given video view is streamed from a location and thenumber of people detected present in that captured space has escalatedor decreased, this allows the consuming participant(s) story view toswitch priority to the widest camera input view available for aspecified duration (in addition to changing which streams are displayed,where necessary). For example, switching from a medium or close-upcropped video view, to a reframed or un-cropped wide view. Thusinforming the consuming meet participants visual awareness of theenter/leave change that occurred in the number of people currentlypresent in the remote location. This priority wide video view behaviourcan be consumed by a single or multi-split view grid in the virtualstory stage environment.

Accordingly, in such embodiments, the system ensures that users are madeaware not only of enter/leave events (or other changes in the visualcontent) of video images that are not being displayed, but also in anyvideo image(s) that are currently being displayed.

The present disclosure allows virtual attendees during virtual meet ups(e.g. video conferences) to track participant enter/leave activity andawareness during the story video view experience.

Note that references to users “entering” or “leaving” in the presentcontext do not refer to new network endpoints connecting to ordisconnecting from a call. Rather, they refer to changes in the visualcontent of a moving image carried by a video stream that is generated bya given endpoint (e.g. a general purpose computer, or dedicatedvideo-conferencing hardware). That is, references to leaving refer to auser who was previously detectable in a moving image becomingundetectable therein (e.g. because he has walked out of a real-woldconference room, or at least out of view of a camera in the conferenceroom), whereas references to joining refers to a user newly becomingvisible in the moving image (e.g. because he has walked into theconference room, or at least walked into view of the camera).Accordingly, references to displaying a previously non-displayed videoimage in response to detecting a user entering or leaving mean that themoving image is displayed in response to a change in the visual contentof the moving image such that the total number of users that aredetectable in that image changes (as opposed to changing which videoimages are displayed based on existing endpoint disconnecting from ornew endpoints connecting to a call).

The enter/leave in video detection can be triggered by various sensortypes, face detection, in video frame or in physical space movementdetection, and additionally in feed audio or in physical space audiodetection depending on the sensors used in some embodiments. Generally,in such embodiments, one or multiple additional sensors may be utilizeddepending on what data is available, in addition to the visual contentof the images themselves. Moreover, other events may also be taken intoaccount in addition to changes in the visual content, such as thedetection of a user checking into a meeting room space via anotherdevice (mobile, laptop . . . ) thus notifying the group they are nowpresent in that physical space. The “wide room video view” from thatspace can be escalated into the story view for group awareness.

In the present disclosure, a duration timer is assigned to non-verbalsingular events. Upon detection of a non-verbal event the video ispromoted and assigned a duration and priority in the active video stackof the live story view sequence during a video call, providing activityawareness of the group non-verbal communication to the remote consumingattendee's video sequence for playback.

The present disclosure allows virtual attendees during live playback ofvideo based meet up experiences to track participant non-verbalcommunication activity and awareness during the story video viewexperience on the stage. A duration is assigned to the non-verbalcommunication priority item for story view experiences, resulting in aright place and right time for the activity to surface in the storyview. This increases participant engagement, activity and spatialawareness of the users, and the people attendance scale of a meet up.

In addition to live video in video calls, the present techniques mayalso be applied to recorded video of such calls at a later time.

Herein, references to users being currently “visible” in a moving image(or similar) carried by a video stream does not necessarily mean thatthe video image is currently being viewed. A user can be visible in amoving image that is not currently being displayed, in the sense thatthey are detectable in the visual content of that image based oncomputer-implemented image processing applied to the moving image, suchas facial or skeleton tracking applied to the image by a computer. Thevisual content of a moving image means information that is conveyed bypixel values of the moving image, and which would thus be conveyed to aviewer were that moving image to be displayed (i.e. played out) to himThus, in accordance with the present techniques, it is ultimatelychanges in those pixel values—and in particular a change in theinformation that is conveyed by the changing pixel values—that causescertain video images to be selected for displaying for appropriatelychosen intervals to convey the information change to one or more callparticipants. Each such change in the information conveyed by the visualcontent of a moving image is referred to individually herein as a“non-verbal singular event”, which includes for example changes in thenumber of users in the moving image and certain expected (i.e.recognizable) movements by a user in the moving image.

A moving image is also referred to herein as a “video image”, and meansa sequence of frames (i.e. static images) to be played out in quicksuccession to give the impression of movement. Unless otherwiseindicated, any references to “an image” below denote a moving image inthis sense, rather than a static image. References to “displaying avideo stream” mean displaying the moving image carried by that videostream.

FIG. 1 shows a communication system 100 comprising a first user 4 a(User A) who is associated with a first user terminal 6 a, a second user4 b (User B) who is associated with a second user terminal 6 b, a thirduser 4 c (User C) associated with a third user terminal 6 c and a fourthuser 4 d (User D) associated with a fourth user terminal 6 d. Each ofthe respective users participating in the video conference may be atdifferent remote locations. A fifth user 4 e (User E) is shown as a userthat may not always be present at the location associated with User B(or at least, may not always be in view of the camera device 7 b).Whilst only five users have been shown in FIG. 1, it will be appreciatedthat the communication system 100 may comprise any number of users andassociated user devices. It will also be appreciated that, whilst eachuser terminal 6 is shown with an associated camera device, 7, one ormore of the user terminals may be associated with one or more additionalcameras or sensors (e.g. microphone array, Kinect etc.), therebyallowing more streams of input from that location. For example, userterminal 6 b is shown to have an additional camera device 9. Theadditional camera device 9 may provide an alternative angle from whichto capture a video image of user 4 b (and/or user 4 e). The multiplecamera devices and/or sensors can be identified as co-located andsourced for the best positioned sensor input in the storied experience.

More generally, one or more peripheral devices, such as externalcameras, audio mics, motion sensors etc. may be connected to thenetwork. These can be checked in or added to a specific parent devicelocation via Bluetooth, Wi-Fi, network login etc. These peripheraldevices may act as added sensors or user preference inputs. Sensorcoverage (i.e. the time at which particular sensors are activated) maybe constrained so as to cover a storied event at the right place andtime. For example, a standard type of stored experience may include“chapters” or “phases”; phases such as “start”, “story”, “end”, “manage”and “relive”. These chapters or phases may be used to manage thepriorities and coverage of behaviour so as to ensure that such behaviouris captured at the appropriate times.

The user terminals 6 a, 6 b, 6 c and 6 d can communicate over thenetwork 2 in the communication system 100, thereby allowing the users 4a, 4 b, 4 c and 4 d to communicate with each other over the network 2.The network 2 may be any suitable network that has the ability toprovide a communication channel between user terminals 6 a, 6 b, 6 c and6 d. For example, the network 2 may be the Internet or another type ofnetwork such as a High data rate mobile network, such as a 3rdgeneration (“3G”) mobile network.

The user terminals 6 a, 6 b, 6 c and 6 d can be any type of user devicesuch as, for example, a mobile phone, a personal digital assistant(“PDA”), a personal computer (“PC”) (including, for example, Windows™,Mac OS™ and Linux™ PCs), a gaming device (Xbox), a group room meetingdevice (e.g. Surface Hub) or other embedded device able to connect tothe network 106. Each user terminal is arranged to receive informationfrom and output information to one or more of the other user terminals.In one embodiment, each user terminal comprises a display such as ascreen and an input device such as a keypad, a touch-screen, cameradevice and/or a microphone.

User terminals 6 a, 6 b, 6 c and 6 d each execute a communication clientapplication provided by a software provider associated with thecommunication system. The communication client application is a softwareprogram executed on a local processor in the respective user terminal.The communication client application performs the processing required atthe respective user terminal in order for each user terminal to transmitand receive video data (carried in the form of video streams) over thenetwork 2. Each user terminal is connected to the network 2.

The communication client application is a video-conferencing applicationthat enables users 4 a, 4 b, 4 c and 4 d to participate in a videoconference. The communication client application provides a meansthrough which each user can share any video data captured at their userdevice (e.g. by an associated camera device, such as those shown at 7 a,7 b, 7 c, 7 d and 9 of FIG. 1) with one or more of the other users. Thecommunication client application also provides a means through whicheach user can receive, at their respective user terminal, the video datacaptured by the other participants of the video conference.

For example, a user, such as user A, may initiate the video conferenceby transmitting a request to one or more other users, such as users B, Cand D. Upon accepting the request from user A, users B, C and D may eachreceive video data from user A, and transmit their own video data toeach of the other users that have agreed to partake in the videoconference. For example, user B may receive the video data captured byone or more of users A, C and D.

Groups of people (i.e. users) may also be detected and identified at asingle location or via single or multiple devices. This is important forimproving group awareness and coverage from a single location into thevirtual storied experience. This also ensures that all of thedistributed people (users) and groups of people (users) are fullyengaged and aware of everyone's presence.

Connected to the network 2 is a control server 102 arranged to receivevideo streams from one or more user terminals (e.g. user terminals 6 a,6 b and 6 c) and to determine one or more other user terminals (e.g.user terminal 6 d) to transmit one or more of the received video streamsto. The control server 102 may be implemented on a single computingdevice. The control server 102 may also operate to support performanceof the relevant operations in a “cloud computing” environment whereby atleast some of the operations may be performed by a plurality ofcomputing devices.

User terminals 6 a, 6 b and 6 c may correspond to user terminal 6 d(which, in the following examples, is described as a “receivingterminal”). The user terminal 6 d executes, on a local processor, acommunication client which corresponds to the communication clientexecuted at the user terminals 6 a, 6 b and 6 c. The client at the userterminal 6 d performs the processing required to allow the user 4 d tocommunicate over the network 2 in the same way that the clients at userterminals 6 a, 6 b and 6 c perform the processing required to allow theusers 4 a, 4 b and 4 c to communicate over the network 2. The userterminals 6 a, 6 b, 6 c and 6 d are end points in the communicationsystem. FIG. 1 shows only five users (4 a, 4 b, 4 c, 4 d and 4 e) andfour user terminals (6 a, 6 b, 6 c and 6 d) for clarity, but many moreusers and user devices may be included in the communication system 100,and may communicate over the communication system 100 using respectivecommunication clients executed on the respective user devices, as isknown in the art.

FIG. 2 illustrates a detailed view of the user terminal 6 on which isexecuted a communication client for communicating over the communicationsystem 100. The user terminal 6 comprises a central processing unit(“CPU”) 202, to which is connected a display 204 such as a screen ortouch screen, input devices such as a keypad 206 and a camera 208. Anoutput audio device 210 (e.g. a speaker) and an input audio device 212(e.g. a microphone) are connected to the CPU 202. One or more additionalsensors (not shown) such as a “Kinect” device or Mixed Reality devicesuch as “Hololens” may also be connected to the CPU 202. The display204, keypad 206, camera 208, output audio device 210, and input audiodevice 212 and additional sensors may be integrated into the userterminal 6 as shown in FIG. 2. In alternative user terminals one or moreof the display 204, the keypad 206, the camera 208, the output audiodevice 210, and the input audio device 212 and the additional sensorsmay not be integrated into the user terminal 6 and may be connected tothe CPU 202 via respective interfaces. One example of such an interfaceis a USB interface. The CPU 202 is connected to a network interface 224such as a modem for communication with the network 106. The networkinterface 224 may be integrated into the user terminal 6 as shown inFIG. 2. In alternative user terminals the network interface 224 is notintegrated into the user terminal 102. The user terminal 102 alsocomprises a memory 226 for storing data as is known in the art. Thememory 226 may be a permanent memory, such as ROM. The memory 226 mayalternatively be a temporary memory, such as RAM.

FIG. 2 also illustrates an operating system (“OS”) 214 executed on theCPU 202. Running on top of the OS 214 is a software stack 216 for thecommunication client application referred to above. The software stackshows an I/O layer 218, a client engine layer 220 and a client userinterface layer (“UI”) 222. Each layer is responsible for specificfunctions. Because each layer usually communicates with two otherlayers, they are regarded as being arranged in a stack as shown in FIG.2. The operating system 214 manages the hardware resources of thecomputer and handles data being transmitted to and from the network 2via the network interface 224. The I/O layer 218 comprises audio and/orvideo codecs which receive incoming encoded streams and decodes them foroutput to speaker 210 and/or display 204 as appropriate, and whichreceive unencoded audio and/or video data from the microphone 212 and/orcamera 208 and encodes them for transmission as streams to otherend-user terminals of the communication system 100. The client enginelayer 220 handles the connection management functions of the VoIP systemas discussed above, such as establishing calls or other connections byserver-based or P2P address look-up and authentication. The clientengine may also be responsible for other secondary functions notdiscussed herein. The client engine layer 220 also communicates with theclient user interface layer 222. The client engine layer 220 may bearranged to control the client user interface layer 222 to presentinformation to the user of the user terminal 200 via the user interfaceof the client which is displayed on the display 204 and to receiveinformation from the user the user terminal 200 via the user interface.

A display module 228 of the UI layer 222 is shown. The display module228 determines the manner in which any video streams received over thenetwork (via the network interface) are displayed at the display of theuser terminal 6. For example, the display module may receive layoutparameters from the network interface, and use these to generate, orselect, a particular layout for displaying the one or more videostreams.

The display module may also receive data relating to the video streamsthemselves, such as, for example an associated priority value. Thedisplay module may use the priority value associated with a video streamto determine the duration for which that video stream shall be displayedat the user terminal 6 and/or where, within a predetermined layout, thevideo stream will be displayed.

FIG. 3 illustrates a more detailed view of the control server 102 shownin FIG. 1.

As can be seen in FIG. 3, the control server comprises a networkinterface 314 for receiving and transmitting video streams from and toother user terminals, over the communications network 2.

FIG. 3 corresponds to the control server of FIG. 1, where users A, B, Cand D are participants of a video conference.

For the sake of conciseness, the control server shown in FIG. 3 isdescribed from the perspective of determining which of the users, usersA, B, C and E, to display to a receiving user, User D. While stream s4(the stream associated with user D) is not shown as an input to thecontrol server, it will be appreciated that stream s4 may also be aninput to the control server, and the control server may determine foreach individual user (i.e. users A, B, C and D), which of the otherusers (and their associated video streams) to display to that user.

In the example shown in FIG. 3, video streams s1, s2 and s3 are receivedat the network interface from user terminals 6 a, 6 b and 6 crespectively (each carrying a moving image of users A, B and Crespectively). As a result of the operations performed by selector 312(described later), streams s1 and s2 are selected and transmitted, viathe network interface, to User D's user terminal 6 d, herein referred tomore generally as the “receiving terminal”.

It should be noted that in alternative embodiments, two or more of thevideo streams may be received from a single camera device (i.e. there isnot necessarily a one-to-one mapping between camera devices and videostreams). In such a case, the video streams may be treated by theselector in the same way as if they had been received from separatedevices.

In the embodiment described in relation to FIG. 3, at least two of thevideo streams are received at the control server from differentinstances of the communication client application, running on differentuser devices. That is, at least two of the video streams are receivedfrom different network endpoints having different network addresses(e.g. different IP addresses, or at least different transportaddresses). For example, each of the video streams may be received froma different user terminal, where each of the different user terminalsexecute an instance of the communication client application (as is thecase with streams s1, s2 and s3 shown in FIG. 3). For example, differentusers may be logged into each of the communication client applicationsusing a username that is different to another user that has logged intoanother instance of the communication client application. In any case,the at least two of the video streams received at the control server arereceived from different instances of the communication clientapplication, and not, for example, from different but co-located cameradevices (i.e. all in a conference room), which may be connected to thenetwork via a single instance of the communication client application.

In other cases, some of the streams may be received at the server fromthe same client. That is, a client may transmit more than one stream tothe server allowing the server to select between different streams fromthe same client in the same manner.

For example, a single camera may stream multiple streams derived from alocally-captured “master” video image. For example, each stream maycarry video image corresponding to a respective part of the master image(e.g. of different regions, different cropping's etc.).

As another example, multiple camera feeds may be streamed via thenetwork from one location to a shared virtual stage experience. If thenumber of people detected as present in that single space has escalatedor decreased, the story view priority may be switched and the widestcamera view and framing available from the total number of active videoinput sensors may be selected. This provides ideal visual awareness ofthe total number people currently present in that specific remotelocation. This priority wide video view behaviour can be consumed by asingle or multi-split view grid in the virtual story stage environment.

The network interface 314 is connected to a feature detection module308, which may for example comprise a skeletal tracking module 308and/or a facial detection module (not shown separately). The skeletaltracking module is configured to identify the skeletons of one or moreusers in one or more of the video streams received at the networkinterface. The skeletal tracking module may use the same process foridentifying skeletons as Microsoft's Kinect sensor. The facial detectionmodule is configured to detect the face(s) of any users in each videostream. In the example shown in FIG. 3, the feature detection module 308receives video streams s1, s2 and s3, and determines whether any users(or rather, skeletons) are present in the respective video streams.

Having identified that one of the video streams is carrying an image ofone or more users, the feature detection module 308 may forwardinformation about the user(s) in the corresponding video stream to auser-number tracking module 318.

The user-number tracking module 318 is configured to receive thisinformation and to detect any changes in the number of users that aredetected in the respective moving images of the one or more videostreams. For the sake of clarity, the following embodiments aredescribed in the context of the user-number tracking module detecting achange in the number of users that are detected in a single videostream, e.g. video stream s2.

The user-number tracking module 318 may detect a change in the numberusers by detecting a change in the number of faces detected in themoving image of a video stream. For example, the facial tracking modulemay be configured to detect a number of face-shaped objects that arepresent in the moving image of the video stream and the user-numbertracking module may be configured to determine when this number changes,or when it is likely that this number will change. For example, theuser-number tracking module 318 may be configured to detect changes inthe movement of a user's face, and to detect when this movementindicates that a user is preparing to leave the location associated withtheir user terminal (or at least, the location that is detected by thecamera device associated with that user terminal). Thus, a receivinguser is able to view the user as they leave, rather than being presentedwith a video stream that, for reasons unknown to them, contains fewerusers.

It is valuable during real-time events to keep live activity as apriority. The replay of past activity should be limited to a durationtimer that starts at the real start of the event. Thus ensuring thatduring sync events the story doesn't create a delayed abstraction ofreality that would disrupt the continuity of the group experience.

In this respect, note that the term “detecting a change in the totalnumber of users in a moving image” encompasses a pre-emptive detection,i.e. the detection of one or more events (such as a user standing up)that indicate a change in the total number of users is likely to occurimminently.

In some embodiments, the user-number tracking module 318 may be limitedto detecting a change in a number of known (i.e. recognised) faces. Forexample, the facial tracking module may be configured to determinewhether the faces identified in the received video stream correspond toone or more known users. This may be performed, for example, byidentifying one or more facial features of the one or more detectedusers, and comparing these with the facial features of known users,stored e.g. in a database in memory at the control server (not shown).The user-number tracking module may then be used to determine whetherany changes in the number of detected faces correspond to an increase ordecrease in the number of known faces. This embodiment may be useful fordetermining when a person of importance is no longer present at a userterminal that is participating in the video conference (which in turn,may affect a priority value associated with the corresponding videostream).

In an alternative embodiment, the user-number tracking module 318 maydetect a change in the number users by detecting a change in the numberof skeletons detected in the moving image of a video stream. Forexample, as the skeletal tracking module of the feature detection modulemay be configured to detect the number of skeletons that are present inthe moving image of the video stream and the user-number tracking modulemay be configured to determine when this number changes, or when it islikely that this number will change. For example, the user-numbertracking module 318 may be configured to determine when the movement ofa user's skeleton is suggestive that the user is about to leave thelocation associated with their user terminal (or at least the locationthat is captured by the associated camera device).

In a further, alternative embodiment, the user-number tracking module318 may detect a change in the number of moving users in threedimensions at a particular location (associated with a user terminal),based on a combination of full body 3 d motion capture, facial detectionand voice recognition. For example, the user-number tracking module maybe configured to receive data from a camera, one or more depth sensorsand a multi-array microphone 212 associated with one of the userterminals transmitting a video stream to the control server. Theuser-number tracking module may then determine whether any changes inthis data corresponds to a change, or potential change, in the number ofusers that are present at the location associated with the user terminaltransmitting the video stream to the control server.

If the user-number tracking module detects that there has been a changein the number of users detected in the moving image of a video stream,the user-number tracking module may provide an indication that a changein the number of users has been detected (herein referred to as theuser-count indicator), to a selector 312. The user-count indicator mayprovide an indication of the change in the number of the users that hasbeen detected—e.g. a numerical value indicating the number of users thathave entered or left the video image of a particular video stream.

Selector 312 is configured to receive each of the plurality of videostreams received at the network interface 314, and to determine which ofthese to cause to be displayed at one or more user terminals. In theexample of FIG. 3, selector 312 is configured to determine which of thevideo streams s1, s2 and s3 to cause to be displayed as User D's userterminal (i.e. the receiving terminal).

Selector 312 is also configured to receive the user-count indicator fromthe user-number tracking module. The user-count indicator enables theselector 312 to determine which of the plurality of received videostreams to select and cause to be displayed at a particular user's userterminal (in this example, user D's user terminal 6 d).

The user-count indicator also enables the selector 312 to determine atime interval for which the one or more video streams associated with adetected change in the number of detected users should be displayed at aparticular receiving terminal (again, in this example, user D's userterminal 6 d).

For example, the selector 312 may use the user-count indicator to querythe entries of a database, where each possible change in the number ofdetected users is mapped to a predetermined time interval. The entriesof such a database are shown in FIG. 4 (discussed later).

Alternatively, the selector 312 may use the user-count indicator todetermine a time interval as and when it is needed. That is, the timeinterval may be a function of the change in the number of detectedusers, and the selector may compute the time interval based on theuser-count indicator. For example, the selector may determine timeintervals such that the larger the detected change in the number ofusers, the longer the time interval for which the associated videostream is to displayed at a receiving terminal.

In some embodiments, a tagged location or device may default to noenter/leave detection (i.e. no detection of a change in the number ofdetected users), while “focus locations” may allow the detection tooccur. This feature may be valuable for a “broadcast or presentation”experience, where, for example, it may be desirable to de-escalate (e.g.assign a lower priority) to the audience enter/leave activity of one ormore remote audiences consuming the broadcast or presentation. Thepresenting user's point view may have a different enter/leave activitypriority than the audience's, such that the presenting user has moreawareness of the enter/leave activity of the one or more remoteaudiences, whilst each remote audience has less awareness of theenter/leave activity of the other remote audiences. A user may tag adevice or particular location as having either no enter/leave detectionor being a “focus location” by configuring their user settings in the“settings” menu of the communication client application.

In further embodiments, the control server (e.g. the user-numbertracking module 318) may tag the entry and exit of users forpost-production or curation of the event. For example, a participant maywish to re-visit the activity or meeting post the real time event. Theenter/leave activity can be used to understand who was involved inspecific activities at specific times. In certain embodiments, thecoverage of join/leave activity of users may be limited, where forexample, a change in the number of detected users is detected multipletimes in a short amount of time (herein referred to as “mass” activity).For a video stream that corresponds to such activity, the selector mayselect a default “mass join, mass leave or mass join/leave” timeinterval for displaying the video stream. This may allow the videostream to be displayed persistently—i.e. for longer than the timeinterval associated with the first detected change in the number ofusers. Alternatively, in response to detecting “mass” activity theselector may cause the associated video stream to not be displayed at areceiving terminal. Whether the video stream is displayed or hidden maydepend on the type of meeting that the participants of the videoconference are partaking in e.g. presentation/speaker or collaboration.

During the detection of mass visual activity in frame, timers may beused to extent or hold the detection duration until the activity calms.This keeps the video switching from becoming too active and holds theuser's attention on the action. This is similar to participant massactivity detection, where, on detection of mass activity, activity isgrouped into one transition instead of a sequence of transitions tosupport a fluid and natural stored sequence of views and video.

The selector 312 may also use the user-count indicator to determine apriority associated with a video stream in which a change in the numberof detected users was detected. For example, video streams in which achange in the number of detected users is detected may be deemed moreworthy of display than others. For example, video streams associatedwith non-speaking or relatively stationary users may be associated witha lower priority value than video streams for which a change in thenumber of associated users has been detected. The priority value may bereset to a new value once the determined time interval (described above)has elapsed.

In a situation where changes in the number of detected users is detectedin multiple video streams, but only a limited number of video streamscan be displayed at a particular receiving terminal, the selector 312may use the priority value associated with each of the video streams todetermine which of the video streams to select for display at thereceiving terminal. Video streams in which a larger change in the numberof detected users has been detected may be assigned a higher priorityvalue than video streams in which a smaller change in the number ofdetected users was detected. This may ensure that video streamsassociated with small but regular changes in the number of detectedusers are not displayed in favour of other video streams, where moredramatic changes in the number of detected users may be occurring.

A user defined priority view may also influence the sensor priorityorder. For example when an installer is setting up a room the admin mayassign a front of room camera as the dominant view for that space,allowing all other device sensors that are or get detected as alternateview option for the story view. An alternate view could be escalated topriority view when the added or reduced user detection is defined as themore coverage view.

The priority value may also determine the manner in which a selectedvideo stream is displayed relative to any other video streams which havealso been selected for display at the receiving terminal (i.e. relativeposition and size).

The selector 312 may have stored in memory, a selection of grid layouts,and the selector 312 may select a particular grid layout for displayingthe one or more selected video streams. The grid selected by theselector 312 may depend on the number of video streams that the selector312 has selected for display at a particular user terminal.

Story continuity will put a priority on assignment to one story gridarea for a specific location during a meetup, allowing wide viewpriority to replace the current view from that location if it is presentin the grid, as opposed to populating another grid as a default. Thisalso influences continuity as common location feeds can be focused intoa single storied frame view when sequenced in the stage grids forconsuming participants (that is, the video streams associated with thesame user location can be displayed at the same, or a similar locationin the grid, even if the grid itself is updated). During a room to roomshared experience or a room to audience broadcast multi cameras viewsdistributed across a multi-frame grid makes more sense to consume as theconsuming participants associate all the views from a single location.

The selector 312 may also use the priority associated with a selectedvideo stream (based e.g. on the detected change in the number ofdetected users) to determine where, within the selected grid layout,that video stream is to be displayed. Some examples of possible gridlayouts are shown in FIG. 9. For example, grid layout 902 may be used todisplay a single video stream, grid layout 904 may be used to displaytwo video streams simultaneously, grid layout 906 may be used to displaythree video streams simultaneously, and grid layout 908 to display fourvideo streams simultaneously. Whilst only four grid layouts are shown inFIG. 9, it will be appreciated that a grid layout may be selected so asto display any number of video streams. For example, a grid layoutcomprising five or more units may be selected to display four selectedvideo streams. Whilst the grid layouts shown in FIG. 9 are all shownwith rectangular units, the units of each grid may be of any shape andare not constrained so as to all be of the same shape.

For greater story continuity and fluid transitions between the differentvideo streams that are displayed at the receiving terminal, the selectormay be configured to ensure that there is a limited duration of time inwhich the units of the selected grid layout can be updated (i.e. a newvideo stream can be selected for display, at that unit of the grid).

For example, in one embodiment, the selector may ensure that only oneunit of the selected grid is changed at a time—i.e. no new video streamsare displayed at any of the other units of the grid, during the secondtime interval.

Alternatively, in a second embodiment, the selector may ensure thatthere is limited duration of time in which multiple units of theselected grid layout can be updated (i.e. to display the video streamsin which a change in the number of users was detected). For example,following e.g. the selection of a first video stream, the selector mayonly allow other units of the selected grid to be updated, if these canbe updated before the limited duration of time elapses.

These embodiments ensure that the least amount of video grid viewupdates occur within a designated duration of time, thereby making it aseasy as possible for users to follow user activity within the “StoriedExperienced View”.

An allowance for a delayed playback of an enter/leave event may beutilized to communicate the clearest story experience. In this case, ifa person walking out of a room is captured on video, the delayedplayback could show the actual exit footage playback followed by thereal-time empty room, thereby giving the greatest intuitive perceptionof events to the remote participants.

The selected grid and positioning of each of selected video streamswithin the grid may be indicated to the receiving user terminal (e.g.terminal 6 d) in the form of layout parameters, as shown in FIG. 3. Thereceiving user terminal may interpret the layout parameters so as todisplay each of the selected video streams at their respective positionsin the selected grid. Three types of time interval (i.e. durations) aredescribed below.

Short Duration:

A set duration attached to a non-verbal event. Short would be set to aspecific duration (example: 1.5 seconds). Allowing the priority assignedvideo to priority stack in the video story for consuming participantsbut not interrupt the active speaking audio signal. The active speakingaudio signal would remain constant. Short will be assigned toparticipant activity that is added awareness but not essential at anextended activity. Including reaction shots: smiling, head nodding, headshaking, hand waving, hand pointing . . . ).

Medium Duration:

A set duration attached to a non-verbal event. Medium would be set to aspecific duration (example: 2.5 seconds). Allowing the priority assignedvideo to priority stack the video story for consuming participants butnot interrupt the active speaking audio signal. The active speakingaudio signal would remain constant. Medium is assigned to specificactivities deemed important to group activity awareness such as a changeof body location in the room. Or a detection of a new body or person inthe room (stand, sit, walk enter, leave a location).

Extended Duration:

The set duration for dominant activity participants. This duration isprimarily assigned to active speaker. Giving active speaker the dominantstory priority unless interrupted by a short duration story view ordepreciated due to lack of speaking. An example of this is if story viewis in single grid view and is an edge to edge video of active speaker.When a short or medium duration priority video is triggered to replacethe active speaker video (but not the active audio) once the limitedduration video has timed out it is replaced by the continuous activespeaker video view that was populated at this location previously.Alternatively, if a “Mass” activity is detected, where multiple triggersare being detected in a short amount of time (e.g. multiple changes inthe number of detected users in a short amount of time), this wouldextend the priority view to avoid multiple view switches from stackingsequentially. This ensures that a user is not disorientated by themultiple view switches that would otherwise occur each time e.g. a newnumber of users is detected.

For greater story continuity and fluid people engagement experiences thecamera view grid updates are populated as a single or a group in a setduration of time whether it be from a single location or multiplelocations. This is to ensure that the video playback is as fluid andnoise-free as possible. It is also to ensure the least amount of videogrid view updates occur within a designated duration of time, therebyallowing the story experience to be as engaging and easy to follow theaction.

Persistent Duration:

The set duration for dominant activity participants. This duration isprimarily assigned to a user pinned or view type that does not allow avideo view to be interrupted. Thus duration is continuous until the userre-assigns the view or the meeting ends.

Non-verbal communication duration priority metric for: body, arm, hand,gesture, head, face and eye movement detection for story video prioritymetric. Duration priority metric works in conjunction with a playbackdurations library: short, medium, priority, and extended specification.As well as story grid location priority designated by a stack ranking ofmost recent activity, participant association or user preference.

Returning to the example illustrated in FIG. 3, the selector 312 mayreceive an indication that a change in the number of detected users wasdetected in streams s1 and s2 and based on this, select streams s1 ands2 for transmission to user 4 d's user terminal. The selector 312 mayselect, for example, grid layout 904, shown in FIG. 9, and forward thecorresponding layout parameters to the receiving terminal. In responseto receiving the layout parameters, the receiving terminal may thenrender the two video streams such that the first video stream, s1, isdisplayed at a first location of the grid (e.g. the left-hand unit ofthe grid), and the second video stream, s2, is displayed at a secondlocation of the grid (e.g. the right-hand unit of the grid). In someembodiments, it may not be necessary to send of all the layoutparameters to the receiving terminal, if for example, there is no changein the number of video streams that are to be displayed at the receivingterminal (as described later in relation to FIGS. 7A and 7B).

Alternatively, the user-count indicator may indicate that a change inthe number of detected users was detected in stream s2 only. Forexample, camera 7 b associated with User B's terminal 6 b may havedetected that another user, User E, has entered or left the locationassociated with User B's user terminal. Based on this, the selector 312may determine to increase the number of video streams displayed at user4 d's user terminal 6 d, by continuing to transmit stream s1 (which wasdisplayed at user 4 d's user terminal 6 d, prior to detecting a reactionfrom User 4B) and also transmitting stream s2 to User D. User D is thusbe able to see that there has been a change in the number of users atUser B's location, in addition to the video of User A, who may be forexample, a speaking user. The control server may transmit layoutparameters for grid layout 904, instead of the layout parameters for902, which were previously used to display User A's video stream (asdescribed later in relation to FIGS. 6A and 6C).

FIG. 4B shows a high-level representation of a database that may be usedby the control server to determine a priority associated with a videostream, based on a detected change in the number of users in the movingimage of the video stream.

As can be seen in FIG. 4B, a first column 402B of the database maycontain entries for each possible change in the number of users detectedin the moving image of a video stream. For example, D1 may correspond tothe number of detected users changing by +1, D2 may correspond to thenumber of detected users changing by +2, D3 may correspond to the numberof detected users changing by +3, D4 may correspond to the number ofdetected users changing by +4, D5 may correspond to the number ofdetected users changing by +5 and so on and so forth. The entries forthe change in the number of detected users may also cover negativevalues, for example, there may be a separate entry for negative changesin the number of detected users (−1, −2, −3, −4, −5 etc.).Alternatively, there may be an entry for each possible change in thenumber of users, regardless of whether it is a positive or negativechange in the number of detected users (e.g. entries for +/−1, +/−2,+/−3, +/−4, +/−5 etc.).

A second column of the database 404B may contain entries for thepriorities associated with each possible change in the number ofdetected users. For example, D1 may have a priority value P1, which ishigher or lower in value than the priority value P2 associated with D2.The priority value may be used to determine the manner in which videostreams are displayed relative to one another. For example, a videostream associated with a higher priority value (e.g. a greater change inthe number of detected users) may be displayed more prominently than avideo stream associated with a lower priority value (e.g. a smallerchange in the number of detected users). The priority values may beused, for example, to determine which of the units of a grid layout(such as those shown in FIG. 9) a selected video stream occupies.

The priorities may also be used to limit the number of video streamsthat are selected for display at a receiving terminal—for example, if achange in the number of detected users is detected in multiple videostreams but only a limited number of video streams can be displayed(effectively) at a particular receiving terminal, the priority valuesmay be used to determine which of those video streams are selected fordisplay.

It will be appreciated that, whilst an individual priority value isshown for each change in the number of detected users (i.e. D1, D2, D3etc.), one or more of the detected changes may share the same priorityvalue and be grouped according to this priority value. For example,there may be a single priority value for a change in the detected numberof users that is equal to or greater than 5. Alternatively, each changein the number of detected users (i.e. D1, D2, D3 etc.) may be associatedwith the same priority value (which in turn, may ensure that each changeis detected for the same time interval—described below). If, forexample, a large number of changes in the number of detected users isdetected over a relatively small time period, a priority value may beselected so as to ensure that the video stream associated with thosechanges is assigned a single priority value (rather than a new value,each time the number of detected users increases or decreases).

A third column of the database 406B may contain entries for the timeinterval associated with each change in the number of detected users,that is, the time interval for which the video stream associated withthat change should be displayed at a receiving terminal. Differentchanges in the number of detected users may be associated with differenttime intervals. For example, a greater change in the number of detectedusers may be associated with a time interval that is greater than asmaller change in the number of detected users. The control server mayuse the time interval to determine when to stop transmitting the videostream associated with the change in the number of detected users, to aparticular receiving terminal.

It will be appreciated that, whilst a separate column is shown for thepriority and time interval of each respective change in the number ofdetected users, these two parameters may in fact be correlated (i.e.derivable from one another).

For example, the priority value associated with a change in the numberof detected users may be used to determine the time interval for whichthe corresponding video stream (or rather, moving image) should bedisplayed. Higher priority values may result in the determination oflonger time intervals. Alternatively, lower priority values may resultin the determination of longer time intervals.

Ultimately, any relationship between the priority value and timeinterval may be used. This relationship may allow time intervals to bedetermined ‘on the fly’ for each detected change in the number ofdetected users. That is, rather than storing a time interval for each ofthe possible changes in the number of detected users in a database, thedatabase may contain entries for the priority values only, and these maybe used to determine the time interval for the change in the number ofdetected users, as and when that change is detected.

One or more other columns 408B of the database may contain entriespertaining to other parameters. For example, these parameters may relateto the grouping of different changes in the number of detected users.

FIG. 5B illustrates a flowchart of the method performed at the controlserver for determining when to select a video stream for display at areceiving terminal (e.g. User D's user terminal), based on a detectionthat there has been a change in the number of users in the moving imageof that video stream

It should be noted that, whilst FIG. 5B only shows a method fordetermining whether to select a single video stream for display at areceiving terminal, the control server may perform multiple instances ofthe described method, e.g. in parallel, in order to determine whether aplurality of video streams should be selected for display at a receivingterminal.

At step, S502, a plurality of video streams are received at the controlserver (i.e. at the network interface of the control server). Forexample, these video streams may be received from the user terminalsassociated with users A, B and C. Alternatively, two or more of thesevideo streams may be received from a single camera device, associatedwith two or more of users A, B and C.

At step S504, the control server selects a subset of the received videostreams for display at the receiving terminal. The control server causesthese video streams to be displayed at the receiving terminal, i.e. bytransmitting them, along with any associated layout parameters, to thereceiving terminal.

At step S506, the control server identifies a video stream that is notcurrently being displayed at the receiving terminal (herein referred toas “the identified video stream”). For example, each of the videostreams received at the control server may include an indication ofwhether or not they are currently being displayed at the receivingterminal. The control server may use these indications to identify avideo stream that is not currently being displayed at the receivingterminal.

Alternatively, a separate module within the control server (not shown inFIG. 3) may keep track of the video streams that were previouslyselected for display at the receiving terminal. This information may beused by the control server to identify a video stream that is notcurrently being displayed at the receiving terminal.

At step S508, the control server identifies one or more features of theone or more users identified within the identified video stream. Asnoted earlier in relation to FIG. 3, the feature detection module 308may detect that a user is present in the identified video stream e.g.based on skeletal and/or facial tracking, by applying suitable skeletaland/or facial tracking algorithms to the moving images respectively.

At step S510B, the control server monitors the number of users that aredetected in the identified video stream, e.g. based on the number facesor skeletons detected by the feature tracking module.

At step S512B, the control server detects a change in the number ofusers detected in the identified video stream, e.g. based on a change inthe number of detected skeletons or faces detected by the user-numbertracking module. As noted earlier, detecting a change in the number ofusers may include determining that a change in the number of users islikely to occur (e.g. based on the movement of a user's skeleton orface).

At step S514, the control server determines whether to cause theidentified video stream to be displayed at the receiving terminal. Ifthe control server determines that the identified video stream shouldnot be displayed at the receiving terminal (indicated by ‘NO’ in FIG.5), the control server continues to monitor the number of users detectedin the identified video stream.

The control server may, for example, determine a priority valueassociated with the detected change in the number of detected users anddetermine whether this value is higher than a priority value determinedfor a second video stream in which a change in the number of detectedusers was also detected. If the priority value determined for the firstvideo stream is lower than the priority value determined for the secondvideo stream, the control server may determine that the first videostream (i.e. the identified video stream) should not be displayed at thereceiving terminal. If, whilst displaying the video stream in which achange in the number of users was detected, the audio of a new speakinguser is detected, the control server may ensure that once the secondtime interval has elapsed, the video stream associated with the newspeaking user is selected for display (and caused to be displayed) atthe receiving terminal.

If the control server determines that the identified video stream shouldbe displayed at the receiving terminal (indicated by ‘YES’ in FIG. 5),the control server selects the video stream for display at the receivingterminal.

At step S516, the control server determines the time interval for whichthe selected video stream should be displayed and any layout parametersthat are needed in order to define the way in which the selected videostream will be displayed at the receiving terminal (e.g. relative to anyother video streams that have been selected for display at the receivingterminal).

In one embodiment, the time interval associated with the selected videostream may be derived, for example, from the priority associated withthe detected change in the number of detected users. As noted earlier inrelation to FIG. 3, each possible change in the number detected usersmay be associated with a priority, and the priority may determine where,and for how long, the selected video stream is displayed at thereceiving terminal.

At step S518, the control server transmits the selected video stream tothe receiving terminal, along with any associated layout parameters. Asnoted earlier, the layout parameters are used by the receiving terminalto determine the manner in which the selected video stream is to bedisplayed.

At step S520, the control server detects that the time intervalassociated with the selected video stream has elapsed and stops sendingthe selected video stream. In response to the time interval elapsing,the control server may transmit new layout data to the receivingterminal, thereby ensuring that screen space is not allocated to videostreams that are no longer being transmitted to the receiving terminal,from the control server. FIG. 6A illustrates an example of a movingimage of a user, user 604A, that may be displayed at the display of UserD's user terminal, during a first time interval, during the videoconference.

FIG. 6A illustrates an example display window 602A that may be used todisplay the moving image of a user 604, to a receiving user, e.g. UserD. User 604 may have been determined to be important based e.g. on arecent detection of the user's speech, or the user having initiated thevideo conference. This user is herein referred to as the “primary user”,with an associated “primary video stream”. In this particular example,user 604 may correspond to e.g. User A in FIG. 1.

At a separate user terminal, a second user, e.g. User B, may bereceiving the primary video stream from User A. The second user may alsobe capturing their own video data (i.e. of themselves) and transmittingthis video data to the control server.

During a first time interval, the control server may determine thatthere has been a change in the number of users detected at the locationassociated with a second user's user terminal (e.g. User B). Forexample, the control server may detect that a third user, User E, hasappeared in the moving image received from User B's user terminal (whichpreviously only contained a moving image of User B). In response todetecting this change in the number of users, the control server mayselect the video stream carrying the image of Users B and E for displayat the receiving user's user terminal (the receiving terminal).

An example embodiment is illustrated in FIG. 6C, where the moving imageof primary user 604 (e.g. User A), has been replaced with the movingimage of users 610 (e.g. User E) and 608 (e.g. User B). As notedearlier, the moving image of users 608 and 610 is displayed for apredetermined time interval (the second time interval). The controlserver may ensure that the primary video stream is not transmitted toUser D, for the duration of this time interval. In FIG. 6C, the videoimage of users 608 and 610 is displayed in display window 602C.

The control server may also ensure that any audio (i.e. detected speech)associated with the primary user is still transmitted to User D. Thatis, the control server may treat the video and audio streams of eachuser (e.g. users A, B and C) separately, and only determine which of thevideo streams (and not audio streams) to select for display User D'suser terminal. Hence, User D is able to continue to listen to the speechof the primary user, whilst also viewing any changes in the number usersat a given user terminal, as and when they occur.

In an alternative embodiment, in response to determining that a thirduser (e.g. User E) has entered or left the location associated with asecond user (e.g. User B), the control server may continue to transmitthe primary video stream to User D's user terminal, and also select thevideo stream associated with the second and third users for transmissionto User D's user terminal. This may also include transmitting new layoutparameters to User D's user terminal 6 d—i.e. layout parameters thatensure that the two video streams are displayed simultaneously usinggrid layout 904 (FIG. 9).

This is shown in FIG. 6E, where both the video streams of the primary(user 604) and the second and third user's (users 608 and 610) are shownsimultaneously, adjacent to one another, in display window 602E. In thisparticular embodiment, User D is able to view both the primary user (whomay be, for example a speaking user) as well as the arrival of (or exitof) User E at User B's location.

FIG. 7A shows an alternative embodiment in which two primary users, user704 and user 706 are displayed at the display of a receiving terminal,during a first time interval, during the video conference. This mayoccur, for example, where both of the primary users are determined asbeing of equal importance (for example, where audio data has beenrecently detected for both users). Alternatively, this may be a defaultsetting for a receiving user that is in a video conference with twoother users (as shown in FIG. 1). User 704 may correspond to e.g. User Aand User 706 may correspond to e.g. User C.

Again, during the first time interval, the control server may detect achange in the number of users detected in the moving image of a receivedvideo stream. For example, the control server may detect that user 710(e.g. User E) has appeared in the video stream that previously carried amoving image of User B only. In response to detecting this change in thenumber of users, the control server may select the corresponding videostream for display at the receiving terminal.

In this particular embodiment, the control server may cause one of thevideo streams displaying a second primary user, user 706, to be replacedwith the video stream associated with the video image in which a changein the number of users was detected. The control server may determine arelative priority of each of the video streams associated with theprimary users (e.g. based on which of the two primary users spoke mostrecently), and based on this, select the video stream with the highestpriority for display at the receiving terminal.

The control server may then continue to transmit the video streamassociated with the highest priority to the receiving terminal, and alsotransmit the video stream associated with the detected change in thenumber of users, to the receiving terminal. This may involve sending newlayout data to the receiving terminal, such that, in response toreceiving the new layout data, the receiving terminal displays the videostream of a first primary user, user 704, as well as users 710 and 712,in a particular arrangement at the receiving terminal.

Such an arrangement is illustrated in FIG. 7C, where the moving image ofprimary user 706, has been replaced with the moving image of users 710and 712 (which again, may correspond to Users B and E). Again, themoving image of users 710 and 712 is displayed for a predetermined timeinterval (the second time interval), which may be independent of thetime interval for which the primary user, user 704, is displayed at thereceiving terminal. In FIG. 7C, the video image of users 704, 710 and712 is displayed in display window 702C.

FIG. 8A shows an alternative embodiment in which three primary users,user 804 (e.g. User A), user 806 (e.g. User F) and user 808 (e.g. UserC) are displayed at the display of the receiving terminal. In thisembodiment, the moving image of a third primary user 808 is replacedwith the moving image of users 812 and 814 (e.g. Users B and E), asshown in FIG. 8C. As described in relation to FIGS. 7A and 7B, each ofthe video streams may be associated with a priority value, and thispriority value may depend on whether the moving image of the respectivevideo stream corresponds to a moving image in which a change in thenumber of users has been detected.

Video streams in which a change in the number of detected users has beendetected may be associated with a higher priority value than a videostream in which no specific user activity has been detected (e.g. nospeech or movement). A moving image with a higher priority value mayoccupy a larger segment of the receiving terminal's display, such asthat occupied by user 804 in FIG. 8A.

In the example of FIG. 8C, the video stream associated with users 812and 814 (e.g. users B and E) is of a high enough priority to replace thevideo of e.g. an inactive user, such as user 808 (e.g. User C) but notof a high enough priority to replace the video of a speaking user, suchas user 804 (e.g. User A).

It will be appreciated that while FIGS. 8A and 8C are described in thecontext of replacing one of three primary video streams with a videostream in which a change in the number of users has been detected, anynumber of the three primary video streams may be replaced with videostreams in which a change in the number of users has been detected(depending on the number of participants in the video conference, andthe number of streams in which a change in the number of users has beendetected etc.).

It will also be appreciated that, if the control server causes anincrease in the number of video streams that are displayed at areceiving terminal, then any number of moving images in which a changein the number of users has been detected may be displayed in addition tothe video streams associated with the one or more primary users.

For example, if a primary user is displayed in a first window 602A, anda change in the number of users is detected in the moving images of twoother video streams, the first window 602A may be updated so as todisplay the video stream of the primary speaker, and the two other videostreams in which a change in the number of users was detected. This mayinvolve transmitting new layout parameters from the control server tothe receiving terminal, e.g. layout parameters that enable the videostreams to be displayed using grid layout 906 instead of grid layout 902(FIG. 9).

For example window 602A may be replaced with a window akin to displaywindow 802C shown in FIG. 8C, with the moving image of each video stream(in which a change in the number of users was detected) being displayedin one of the two smaller segments of the grid layout used forgenerating display window 802C. These two video streams may be displayedfor the same or different time intervals, depending on how similar ordifferent the detected change in the number of users is for each videostream.

In an alternative embodiment, the control server may increase the numberof video streams that are displayed at a receiving terminal, such that adisplay window showing two primary users (e.g. the display window shownin FIG. 7A) is updated so as to also display the video stream associatedwith a change in the number of detected users (e.g. the display windowshown in FIG. 8C), in addition to the video streams of the two primaryusers.

In addition to the above described techniques, signature bodymovement(s) and/or gesture(s) may trigger a camera view change in thevideo stage sequence and framing. An example gesture could be acombination of outward hand movement with a boxed finger framing gestureas a director or cinematographer would do on location for filming Orcould be a single hand “bloom” gesture depending on the activity andcontext of the active experience. Alternatively or in addition to this,a vocal phrase such as, “Skype, Wide View” may trigger a certain streamassociated with the voice input to be displayed to a particular user. Avoice or gesture input may be provided that user themselves, or one ofthe other users on the call allowing users to in some cases explicitlycontrol how the story view evolves for their own benefit, or for thebenefit of other user(s).

In the embodiments described above the change in visual content has beendescribed in relation to detecting a change in the number of users thatare present in the moving image of a video stream. However,alternatively or additionally, a change in visual content may bedetected based on an identification that a user has performed anexpected movement. This embodiment, and variations thereof, aredescribed below. It should be noted that in this embodiment, the samecommunication system 100 and user terminal(s) 6 as described previouslyin relation to FIGS. 1 and 2 is (are) used. This embodiment is thusdescribed below in relation to FIG. 3 onwards.

Returning to FIG. 3, a feature tracking module is also shown having aninput connected to the feature detection module 308 and an outputconnected to the selector 312.

Having identified that one of the video streams is carrying an image ofa user, the skeletal tracking module of the feature detection module 308may forward information about the detected user in the correspondingvideo stream to a feature tracking module 310. This information maycomprise an indication of where the “skeleton” of the user wasidentified within the moving image, for example corresponding topredetermined points on the user's body, e.g. corresponding to knownskeletal points. Either way, this allows the feature tracking module 310to identify particular human features within the moving image. Forexample, the identified “skeleton” of the user may provide a referencefrom which the feature tracking module can identify and track themovement of one or more human features. Alternatively or additionally,the facial detection module may provide information about the detectedface(s) to the feature tracking module 310, allowing the latter to trackthe corresponding facial movements.

Human features may include, for example, the arm, hands, and/or face ofa user. Human features may also include more specific human featuressuch as the eyes, mouth and nose of a user. By tracking the movement ofthese features over the time, the feature tracking module 310 is able todetect and distinguish between different types of reaction that anidentified user is having. For example, the feature tracking module 310may be able to identify user reactions such as: smiling, laughing,frowning, gasping, head nodding, head shaking, hand waving, handpointing, clapping, giving a thumbs up, raising or lowering their arms,celebrating with e.g. clenched fists etc.

The feature tracking module 310 may identify a user's reaction bycomparing the movement of one or more identified human features with theentries of a database 304 storing predetermined, i.e. expected movementsof the corresponding human features. The database of expected movements304 may be stored in memory 302 at the control server.

For example, each expected movement may be defined by a set ofparameters describing the movement of one or more human features. Thefeature tracking module 310 may determine the parameters describing themovement of one or more human features of an identified user and comparethese to the parameters describing known, i.e. expected movements, todetermine whether the user has performed an expected movement.

If the feature tracking module 310 determines that the identified user'smovement of one or more human features corresponds to one of theexpected movements in the database 304 the feature tracking module 310may provide an indication that the expected movement has been detected,to a selector 312.

Selector 312 is configured to receive each of the plurality of videostreams received at the network interface 314, and to determine which ofthese to cause to be displayed at one or more user terminals. In theexample of FIG. 3, selector 312 is configured to determine which of thevideo streams s1, s2 and s3 to cause to be displayed as User D's userterminal (i.e. the receiving terminal).

Selector 312 is also configured to receive an indication from featuretracking module 310 of any expected movements, i.e. reactions that havebeen detected in any of the video streams received at the selector 312.This indication is herein referred to as the “reaction indicator”.

The reaction indicator may inform the selector 312 of any reactions”(i.e. expected movements) that were identified in one or more of thevideo streams received at the selector 312. This enables the selector312 to determine which of the plurality of received video streams toselect for display at a particular user's user terminal (in thisexample, user D's user terminal 6 d).

The reaction indicator also enables the selector 312 to determine a timeinterval for which the video stream associated with that reaction shouldbe displayed at a particular receiving terminal (again, in this example,user D's user terminal 6 d). For example, the selector 312 may use thereaction indicator to query the entries of a database storing a list ofpre-determined reactions and the time intervals for which thosereactions should be displayed at a receiving terminal. The entries ofsuch a database are shown in FIG. 4A (discussed later).

The selector 312 may for example, use the time interval associated withan identified reaction to determine the duration for which a selectedvideo stream should be transmitted to a particular receiving terminal(e.g. user D's user terminal 6 d).

The selector 312 may also use the reaction indicator to determine apriority associated with the identified reaction. For example, certainreactions may be deemed more worthy of display than others, and this maybe indicated in the associated priority value (i.e. the higher thepriority value, the more likely it is that the associated video streamis selected for display).

In a situation where reactions are detected in multiple video streams,but only a limited number of video streams can be displayed at aparticular receiving terminal, the selector 312 may use the priorityvalue associated with each of the detected reactions to determine whichof the associated video streams to select for display at the receivingterminal.

The priority value may also determine the manner in which a selectedvideo stream is displayed relative to any other video streams that arealso selected for display at the receiving terminal (i.e. relativeposition and size).

Having determined which of the plurality of video streams to display atthe receiving terminal (e.g. user D), the selector 312 may also select aparticular layout for displaying the one or more selected video streams(streams s1 and s2 in FIG. 3).

The selector 312 may have stored in memory, a selection of grid layouts,and the selector 312 may select a particular grid layout for displayingthe one or more selected video streams. The grid selected by theselector 312 may depend on the number of video streams that the selector312 has selected for display at a particular user terminal. The selector312 may have stored in memory, a selection of grid layouts, and theselector 312 may select a particular grid layout for displaying the oneor more selected video streams. The grid selected by the selector 312may depend on the number of video streams that the selector 312 hasselected for display at a particular user terminal. The moving images ofthe selected video streams may need to be cropped so as to be displayedat a particular location in the grid. For example, the one or moremoving images may be cropped so as to display the most importantinformation. The moving images may be cropped according to a tight,medium or wide view, depending on the detected expected movement and theselected grid layout.

The selector 312 may also use the priority associated with the reactionidentified in a selected video stream (based e.g. on whether a reactingor speaking user was detected) to determine where, within the selectedgrid layout, that video stream is to be displayed. Some examples ofpossible grid layouts are shown in FIG. 9. For example, grid layout 902may be used to display a single video stream, grid layout 904 may beused to display two video streams simultaneously and grid layout 906 maybe used to display three video streams simultaneously. Whilst only threegrid layouts are shown in FIG. 9, it will be appreciated that a gridlayout may be selected so as to display any number of video streams. Forexample, a grid layout comprising four units may be selected to displayfour selected video streams. Whilst the grid layouts shown in FIG. 9 areall shown with rectangular units, the units of each grid may be of anyshape and are not constrained so as to all be of the same shape.

For greater story continuity and fluid transitions between the differentvideo streams that are displayed at the receiving terminal, the selectormay be configured to ensure that there is a limited duration of time inwhich the units of the selected grid layout can be updated (i.e. a newvideo stream can be selected for display, at that unit of the grid).

For example, in one embodiment, the selector may ensure that only oneunit of the selected grid is changed at a time—i.e. no new video streamsare displayed at any of the other units of the grid, during the secondtime interval.

Alternatively, in a second embodiment, the selector may ensure thatthere is limited duration of time in which multiple units of theselected grid layout can be updated (i.e. to display the video streamsin which a change in the number of users was detected). For example,following e.g. the selection of a first video stream, the selector mayonly allow other units of the selected grid to be updated, if these canbe updated before the limited duration of time elapses.

These embodiments ensure that the least amount of video grid viewupdates occur within a designated duration of time, thereby making it aseasy as possible for users to follow user activity within the StoriedExperienced View. The selected grid and positioning of each of selectedvideo streams within the grid may be indicated to the receiving userterminal (e.g. terminal 6 d) in the form of layout parameters, as shownin FIG. 3. The receiving user terminal may interpret the layoutparameters so as to display each of the selected video streams at theirrespective positions in the selected grid.

For example, referring to FIG. 3, the selector 312 may receive anindication that reactions were detected in streams s1 and s2 and basedon this, select streams s1 and s2 for transmission to user 4 d's userterminal. The selector 312 may select, for example, grid layout 904,shown in FIG. 9, and forward the corresponding layout parameters to thereceiving terminal. In response to receiving the layout parameters, thereceiving terminal may then render the two video streams such that thefirst video stream, s1, is displayed at a first location of the grid(e.g. the left-hand unit of the grid), and the second video stream, s2,is displayed at a second location of the grid (e.g. the right-hand unitof the grid). In some embodiments, it may not be necessary to send ofall the layout parameters to the receiving terminal, if for example,there is no change in the number of video streams that are to bedisplayed at the receiving terminal (as described later in relation toFIGS. 7A and 7B).

Alternatively, the reaction indicator may indicate that a reaction wasdetected in stream s2 only. Based on this, the selector 312 maydetermine to increase the number of video streams displayed at user 4d's user terminal 6 d, by continuing to transmit stream s1 (which wasdisplayed at user 4 d's user terminal 6 d, prior to detecting a reactionfrom User 4B) and also transmitting stream s2 to User D. User D is thusbe able to view the reaction of User B, in addition to the video of UserA. In this particular example, user A may be for example, a speakinguser, while user B is a reacting user, reacting to user A's speech. Thecontrol server may transmit layout parameters for grid layout 904,instead of the layout parameters for 902, which were previously used todisplay User A's video stream (as described later in relation to FIGS.6A and 6C).

Continuity is important for the storied experience; if an event istagged as relating to a certain location it may replace that location'scurrent video stream location in the grid for the new duration timedevent (i.e. the second time interval), whereas a newly promoted eventmay occupy an added grid location or grid escalation.

Stylized grid, duration and location playback may have unique rules forunique circumstances. For example, an end of meeting “montage” coulddisplay a series of related and unrelated events next to each other inthe grid as a stylized reprise of the meeting event. For example, theduration timer for each event could be aligned or intentionally rhythmicto an audio track.

FIG. 4A shows a high-level representation of a database that may be usedby the control server to determine a priority associated with a reactionidentified in one or more of the received video streams. As can be seenin FIG. 4A, a first column 402 of the database may contain entries foreach expected “movement” (i.e. reactions). For example, M1 maycorrespond to “smiling”, M2 may correspond to “head nodding”, M3 maycorrespond to “head shaking”, and so on and so forth.

A second column of the database 404 may contain entries for thepriorities associated with each expected movement. For example, movementM1 (e.g. smiling) may have a priority value P1, which is higher or lowerin value than the priority value P2 associated with movement M2 (e.g.head nodding). The priority values of each respective movement may beused to determine the manner in which video streams are displayedrelative to one another. For example, a video stream featuring ahigher-priority reaction may be displayed more prominently than a videostream featuring a lower-priority reaction. The priority values may beused, for example, to determine which of the units of a grid layout(such as those shown in FIG. 9) a selected video stream occupies.

The priorities may also be used to limit the number of video streamsthat are selected for display at a receiving terminal—for example, ifreactions are detected in multiple video streams but only a limitednumber of video streams can be displayed (effectively) at a particularreceiving terminal, the priority values may be used to determine whichof those video streams are selected for display.

In certain embodiments, there may be a limit on the number of detectiontypes that can occur within a certain duration of time, i.e. to controlthe amount of coverage that is displayed to a user within a specificduration of time. Over coverage of user activity may become disorientingto the user viewing it; it is therefore important to strike the balancebetween expanding the storied awareness of user activity whilst guardingagainst over coverage.

It will be appreciated that, whilst an individual priority value isshown for each expected movement, several movements may share the samepriority value and be grouped according to this priority value. Forexample, rather than having a priority value associated with eachmovement, movements may be grouped according to e.g. the type ofmovement, and movements of the same “type” may share the same priorityvalue. The “type” of a particular movement may determine itscorresponding priority value.

A third column of the database 406A may contain entries for the timeinterval associated with each movement, that is, the time interval forwhich the video stream associated with that movement should be displayedat a receiving terminal Different expected movements may be associatedwith different time intervals depending on the nature of the movement.For example, a movement that involves the movement of the whole of auser's body may have a time interval that is longer than a movement thatcorresponds to e.g. “smiling”. The control server may use the timeinterval to determine when to stop transmitting the video streamassociated with a particular movement, to a particular receivingterminal. Three types of time interval (i.e. durations) are describedbelow.

Short Duration:

A set duration attached to a non-verbal event. Short would be set to aspecific duration (example: 1.5 seconds). Allowing the priority assignedvideo to priority stack in the video story for consuming participantsbut not interrupt the active speaking audio signal. The active speakingaudio signal would remain constant. Short will be assigned toparticipant activity that is added awareness but not essential at anextended activity. Including reaction shots: smiling, head nodding, headshaking, hand waving, hand pointing . . . ).

Medium Duration:

A set duration attached to a non-verbal event. Medium would be set to aspecific duration (example: 2.5 seconds). Allowing the priority assignedvideo to priority stack the video story for consuming participants butnot interrupt the active speaking audio signal. The active speakingaudio signal would remain constant. Medium is assigned to specificactivities deemed important to group activity awareness such as a changeof body location in the room. Or a detection of a new body or person inthe room (stand, sit, walk enter, leave a location).

Extended Duration:

The set duration for dominant activity participants. This duration isprimarily assigned to active speaker. Giving active speaker the dominantstory priority unless interrupted by a short duration story view ordepreciated due to lack of speaking. An example of this is if story viewis in single grid view and is an edge to edge video of active speaker.When a short or medium duration priority video is triggered to replacethe active speaker video (but not the active audio) once the limitedduration video has timed out it is replaced by the continuous activespeaker video view that was populated at this location previously.

Persistent Duration:

The set duration for dominant activity participants. This duration isprimarily assigned to a user pinned or view type that does not allow avideo view to be interrupted. Thus duration is continuous until the userre-assigns the view or the meeting ends.

Non-verbal communication duration priority metric for: body, arm, hand,gesture, head, face and eye movement detection for story video prioritymetric. Duration priority metric works in conjunction with a playbackdurations library: short, medium, priority, and extended specification.As well as story grid location priority designated by a stack ranking ofmost recent activity, participant association or user preference.

For greater story continuity and fluid people engagement experiences thecamera view grid updates should also be populated as a single or a groupin a set duration of time whether it be from a single location ormultiple locations. This is to ensure that the video playback is asfluid and noise-free as possible. It is also to ensure the least amountof video grid view updates occur within a designated duration of time,thereby allowing the story experience to be as engaging and easy tofollow the action.

It should also be noted that, for the durations described above, user orparticipant tagging may also influence the system priority stack. Forexample, a user may tag sensor data (video views) and a priority may beplaced on those views for real-time story playback, recording, orediting after the event.

It will be appreciated that, whilst a separate column is shown in FIG. 4for the priority and time interval of each respective movement, thesetwo parameters may in fact be correlated (i.e. derivable from oneanother).

For example, the priority value of an expected movement may alsodetermine the time interval for which it is displayed. For example, anexpected movement with a higher priority value may be displayed forlonger than an expected movement with a lower priority value.Alternatively, an expected movement with a lower-priority value may bedisplayed for a longer time interval.

Ultimately, any relationship between the priority value and timeinterval may be used. This relationship may allow time intervals to bedetermined ‘on the fly’ for each identified expected movement. That is,rather than storing a time interval for each of the possible expectedmovements in a database, the database may contain entries for thepriority values only, and use these to determine the time intervalassociated with a particular movement, as and when that movement isidentified within a particular video stream.

One or more other columns 408A of the database may contain entriespertaining to other parameters. For example, these parameters may relateto the grouping of different types of reactions, e.g. reactionsinvolving hand movements may belong to a particular group, whilstreactions involving changes in a user's facial expression may belong toa different group. Each expected movement may be associated with a groupvalue and expected movements sharing the same group value may be deemedto be of the same “type” (which may indicate that they share the samepriority values and/or time intervals).

Additionally, the database may include a column for the parametersdefining each expected movement. These parameters may define, for eachexpected movement, the corresponding changes in the relative positioningof a user's eyes, nose, mouth, eyebrows, hands etc. These parameters mayalso be associated with a margin of error—i.e. a range in which therelative positioning of a user's eyes, nose, mouth, eyebrows, hands etc.may change, and still be identified as corresponding to the respectiveexpected movement.

FIG. 5A illustrates a flowchart of the method performed at the controlserver for determining when to select a video stream for display at areceiving terminal (e.g. User D's user terminal), based on anidentification of an expected movement in the moving image of that videostream.

It should be noted that, whilst FIG. 5A only shows a method fordetermining whether to select a single video stream for display at areceiving terminal, the control server may perform multiple instances ofthe described method, e.g. in parallel, in order to determine whether aplurality of video streams should be selected for display at a receivingterminal.

At step, S502, a plurality of video streams are received at the controlserver (i.e. at the network interface of the control server). Forexample, these video streams may be received from the user terminalsassociated with users A, B and C. Alternatively, two or more of thesevideo streams may be received from a single camera device, associatedwith two or more of users A, B and C.

At step S504, the control server selects a subset of the received videostreams for display at the receiving terminal. The control server causesthese video streams to be displayed at the receiving terminal, i.e. bytransmitting them, along with any associated layout parameters, to thereceiving terminal.

At step S506, the control server identifies a video stream that is notcurrently being displayed at the receiving terminal (herein referred toas “the identified video stream”). For example, each of the videostreams received at the control server may include an indication ofwhether or not they are currently being displayed at the receivingterminal. The control server may use these indications to identify avideo stream that is not currently being displayed at the receivingterminal.

Alternatively, a separate module within the control server (not shown inFIG. 3) may keep track of the video streams that were previouslyselected for display at the receiving terminal. This information may beused by the control server to identify a video stream that is notcurrently being displayed at the receiving terminal.

At step S508, the control server identifies one or more human featuresof the user identified within the identified video stream. As notedearlier in relation to FIG. 3, the feature detection module 308 mayidentify that a user is present in the identified video stream (e.g.based on skeletal and/or facial tracking) and a feature tracking module310 may use this information to identify one or more human features ofthe identified user.

At step S510A, the control server tracks the movement of the one or moreidentified human features. This may involve, for example, tracking themovement of a user's eyes and mouth, to determine whether the user issmiling or frowning etc.

At step S512A, the control server identifies that the movement of theone or more human features corresponds to an expected movement, i.e. aknown “reaction”. As noted earlier in relation FIG. 3, this may involvedetermining parameters for the identified movement and comparing thesewith the parameters defining expected movements.

At step S514, the control server determines whether to cause theidentified video stream to be displayed at the receiving terminal. Ifthe control server determines that the identified video stream shouldnot be displayed at the receiving terminal (indicated by ‘NO’ in FIG.5), the control server continues to track the one or more identifiedhuman features of the user identified in the identified stream.

The control server may, for example, determine a priority valueassociated with the identified movement, and determine whether thisvalue is higher than a priority value determined for a second videostream in which an expected movement was also identified. If, forexample, the priority value of the expected movement in the identifiedstream is lower than the expected movement detected in the second videostream, the control server may determine that the identified videostream should not be displayed at the receiving terminal. If, whilstdisplaying the video stream in which an expected movement was detected,the audio of a new speaking user is detected, the control server mayensure that once the second time interval has elapsed, the video streamassociated with the new speaking user is selected for display (andcaused to be displayed) at the receiving terminal.

If the control server determines that the identified video stream shouldbe displayed at the receiving terminal (indicated by ‘YES’ in FIG. 5),the control server selects the video stream for display at the receivingterminal.

At step S516, the control server determines the time interval for whichthe selected video stream should be displayed and any layout parametersthat are needed in order to define the way in which the selected videostream will be displayed at the receiving terminal (e.g. relative to anyother video streams that have been selected for display at the receivingterminal).

In one embodiment, the time interval associated with the selected videostream may be derived, for example, from the priority associated withthe identified “expected movement”. As noted earlier in relation to FIG.3, each of the “expected movements” may be associated with a priority,and the priority may determine where, and for how long, the selectedvideo stream is displayed at the receiving terminal.

At step S518, the control server transmits the selected video stream tothe receiving terminal, along with any associated layout parameters. Asnoted earlier, the layout parameters are used by the receiving terminalto determine the manner in which the selected video stream is to bedisplayed.

At step S520, the control server detects that the time intervalassociated with the selected video stream has elapsed and stops sendingthe selected video stream. In response to the time interval elapsing,the control server may transmit new layout data to the receivingterminal, thereby ensuring that screen space is not allocated to videostreams that are no longer being transmitted to the receiving terminal,from the control server.

FIG. 6A illustrates an example of a moving image of a user, user 604,that may be displayed at the display of User D's user terminal, during afirst time interval, during the video conference.

User 604 may be a user that has been determined to be important, basede.g. on a recent detection of the user's speech, or the user havinginitiated the video conference. This user is herein referred to as the“primary user”, with an associated “primary video stream”.

During the first time interval, the control server may determine that asecond user has reacted to the actions performed by primary user 604.For example, the control server may identify that a second user, hereinreferred to as the “reacting user” 606, has smiled during the first timeinterval. In response to detecting the reacting user's smile, thecontrol server may select the video stream associated with the reactinguser for display at the receiving user's user terminal. This videostream is herein referred to as the “reacting user's video stream”.

An example embodiment is illustrated in FIG. 6B, where the moving imageof primary user 604, has been replaced with the moving image of reactinguser 606. As noted earlier, the moving image of the reacting user isdisplayed for a predetermined time interval (the second time interval).The control server may ensure that the video of the primary user is nottransmitted to User D, for the duration of this time interval.

The control server may also ensure that any audio (i.e. detected speech)associated with the primary user is still transmitted to User D. Thatis, the control server may treat the video and audio streams of eachuser (e.g. users A, B and C) separately, and only determine which of thevideo streams (and not audio streams) to select for display User D'suser terminal. Hence, User D is able to continue to listen to the speechof the primary user, whilst also viewing the reactions of other users,as and when they occur.

When a single grid video view is streaming from a location and a newvideo priority type is detected, a duration type is assigned to thatvideo depending on the detection type and it replaces the lower priorityvideo stream. In most cases non-verbal communication is a video durationpriority only. The audio priority stack preforms separately.

In an alternative embodiment, in response to determining that a seconduser has reacted to the actions performed by primary user 604, thecontrol server may continue to transmit the primary video stream to UserD's user terminal, and also select the reacting user's video stream fortransmission to (and subsequent display at) User D's user terminal. Thismay also include transmitting new layout parameters to User D's userterminal 6 d—i.e. layout parameters that ensure that the two videostreams are displayed using grid layout 904 (FIG. 9).

This is shown in FIG. 6D, where both the video streams of the primaryand reacting users are shown simultaneously, adjacent to one another, atthe display of User D's user terminal. In this particular embodiment,User D is able to view both the primary user (who may be, for example aspeaking user) as well as the reaction of user 606 (who may be reactingto what the primary user is saying).

FIG. 7A shows an alternative embodiment in which two primary users maybe displayed at the display of the receiving terminal, during a firsttime interval, during the video conference. This may occur, for example,where both of the primary users are determined as being of equalimportance (for example, where audio data has been recently detected forboth users). Alternatively, this may be a default setting for areceiving user that is in a video conference with two other users (asshown in FIG. 1).

Again, during the first time interval, the control server may identifythat a third user—the reacting user, has smiled during the first timeinterval. In response to detecting the reacting user's smile, thecontrol server may select the reacting user's video stream for displayat the receiving terminal.

In this particular embodiment, the control server may cause one of thevideo streams displaying a second primary user, user 706, to be replacedwith the video stream associated with the reacting user, 708. Thecontrol server may determine a relative priority of each of the videostreams associated with the primary users (e.g. based on which of thetwo primary users spoke most recently), and based on this, select thevideo stream with the highest priority for display at the receivingterminal.

The control server may then continue to transmit the video streamassociated with the highest priority to the receiving terminal, and alsotransmit the reacting user's video stream to the receiving terminal.This may involve sending new layout data to the receiving terminal, suchthat, in response to receiving the new layout data, the receivingterminal displays the video stream of a first primary user, user 704,and the reacting user, 708, in a particular arrangement at the receivingterminal.

Such an arrangement is illustrated in FIG. 7B, where the moving image ofprimary user 706, has been replaced with the moving image of reactinguser 708. Again, the moving image of the reacting user is displayed fora predetermined time interval (the second time interval), which may beindependent of the time interval for which the primary user, user 704,is displayed at the receiving terminal.

FIG. 8 shows an alternative embodiment in which three primary users aredisplayed at the display of the receiving user's user terminal. In thisembodiment, the video stream of a third primary user 808 is replacedwith the video stream of a reacting user 810. As in FIGS. 7A and 7B,each of the video streams may be associated with a priority, and thevideo stream with the lowest priority may be replaced with the videostream associated with the reacting user. Additionally, the reactinguser's video stream may occupy a larger segment of the receivingterminal's display, depending on the priority associated with theidentified reaction.

In the example of FIG. 8, the detected smile of the reacting user is ofa high enough priority to replace the video of e.g. an inactive user,but not of a high enough priority to replace the video of a speakinguser, such as user 804.

It will be appreciated that while FIG. 8 is described in the context ofreplacing one of three primary video streams with a reacting user'svideo stream, any number of the three primary video streams may bereplaced with the video streams of a reacting user (depending on thenumber of participants in the video conference, and the number reactingusers etc.).

It will also be appreciated that, if the control server causes anincrease in the number of video streams that are displayed at areceiving terminal, then any number of reacting users may be displayedin addition to the one or more primary users.

For example, if a primary user is displayed in a first window 602A, anda reaction is identified in the video streams of two other users, thefirst window 602A may be updated so as to display the video stream ofthe primary speaker, as well as the video streams of the two otherreacting users. This may involve transmitting new layout parameters fromthe control server to the receiving terminal, e.g. layout parametersthat enable the video streams to be displayed using grid layout 906(FIG. 9) instead of grid layout 902 (FIG. 9).

For example window 602A may be replaced with a window akin the displaywindow 802B shown in FIG. 8B, but with a reacting user displayed in eachof the two smaller segments of the display window. The two reactinguser's video streams may be displayed for the same or different timeintervals, depending on the reaction identified in each of the videostreams (e.g. whether they both belong to a group of reactions thatshare the same or similar time intervals).

In an alternative embodiment, the control server may increase the numberof video streams that are displayed at a receiving terminal, such that adisplay window showing two primary users (e.g. the display window shownin FIG. 7A) is updated so as to also display the video stream of areacting user (e.g. the display window shown in FIG. 8B), in addition tothe video streams of the two primary users.

When a multi-grid video story view is streaming for group activity and anew video priority type is detected, a duration type is assigned to thatnew video depending on the detection type. The duration type determineshow long the singular priority will last until the priority is reset tothe current detected participant activity or is overruled by a higherpriority video. In a multi-grid scenario, the least active video isreplaced by the new duration type priority video. Unless it is tagged asrelated to a specific location or participant. Then it replaces the gridview of that same participant or location feed for only the specifiedtime to maintain story continuity.

Generally, unless otherwise indicated, any of the functions describedherein can be implemented using software, firmware, hardware (e.g.,fixed logic circuitry), or a combination of these implementations. Theterms “module,” “functionality,” “component” and “logic” as used hereingenerally represent software, firmware, hardware, or a combinationthereof. In the case of a software implementation, the module,functionality, or logic represents program code that performs specifiedtasks when executed on a processor (e.g. CPU or CPUs). The program codecan be stored in one or more computer readable memory devices. Thefeatures of the techniques described below are platform-independent,meaning that the techniques may be implemented on a variety ofcommercial computing platforms having a variety of processors.

For example, the user terminals may also include an entity (e.g.software) that causes hardware of the user terminals to performoperations, e.g., processors functional blocks, and so on. For example,the user terminals may include a computer-readable medium that may beconfigured to maintain instructions that cause the user terminals, andmore particularly the operating system and associated hardware of theuser terminals to perform operations. Thus, the instructions function toconfigure the operating system and associated hardware to perform theoperations and in this way result in transformation of the operatingsystem and associated hardware to perform functions. The instructionsmay be provided by the computer-readable medium to the user terminalsthrough a variety of different configurations.

One such configuration of a computer-readable medium is signal bearingmedium and thus is configured to transmit the instructions (e.g. as acarrier wave) to the computing device, such as via a network. Thecomputer-readable medium may also be configured as a computer-readablestorage medium and thus is not a signal bearing medium. Examples of acomputer-readable storage medium include a random-access memory (RAM),read-only memory (ROM), an optical disc, flash memory, hard disk memory,and other memory devices that may us magnetic, optical, and othertechniques to store instructions and other data

According to a first aspect, the subject-matter of the presentapplication provides a method effecting a communication event between afirst user and one or more second users via a communication network, themethod comprising implementing on a computer connected to the network:receiving, via the network, a plurality of video streams, each carryinga respective moving image of one or more users; causing the respectivemoving image of a first of the video streams to be displayed at a userdevice of the first user for a first time interval; detecting a changein the visual content of the respective moving image of a second of thevideo streams that is not being displayed at the user device during thefirst time interval; and in response to detecting said change in thevisual content, causing the respective moving image of at least thesecond stream to be displayed at the user device for a second timeinterval.

In embodiments, the change in visual content may be a change in thetotal number of users in the moving image.

The computer may determine the duration of the second time intervalbased on the change in the visual content detected in the moving imageof the second video stream. This change may be detected by the computerapplying a facial detection algorithm to the respective moving image ofthe second video stream to detect a change in the total number of facesin the moving image of the second video stream during the first timeinterval.

Additionally, or alternatively, the change in the total number of usersmay detected by the computer applying a skeletal tracking algorithm tothe respective moving image of the second video stream to detect achange in the total number of skeletons in the moving image of thesecond video stream during the first time interval.

The computer may assign a first priority value to the second videostream for the first time interval, and in response to detecting thechange in the visual content, assign a second priority value denoting ahigher priority to the stream; and the computer may use the secondpriority value to select the second stream from the plurality of videostreams for said displaying at the user device for the second timeinterval.

In some embodiments, causing at least the respective moving image of thesecond video stream to be displayed at the user device may comprisecausing the respective moving image of the first video stream to bereplaced with that of the second video stream, such that the respectivemoving image of first video stream is not displayed at the user devicefor the second time interval.

In alternative embodiments, both the moving images of the first andsecond video streams may be displayed simultaneously at the user devicefor the second time interval.

The computer may be separate from the user device and the computer maycause the moving image of each of the first and second video streams tobe displayed at the user device, by transmitting that stream to the userdevice via the network for displaying thereat.

In further embodiments, the respective moving image of a third of thevideo streams may be displayed at the user device in the first timeinterval in addition to the first video stream, and the computer maycause the respective moving image of the third video stream to bereplaced with that of the second video stream, such that the respectivemoving image of the third video stream is not displayed at the userdevice for the second time interval.

The method according to the first aspect may also comprise: in responseto detecting said change in the visual content, selecting one of aplurality of predetermined layouts for displaying the respective movingimage of the second video stream at the user device for the second timeinterval, wherein each of the plurality of predetermined layouts is fordisplaying a different number of video streams at the user device,wherein a different one of the predetermined layouts is used to displaythe respective moving image of the first stream in the first timeinterval.

In some embodiments, at least two of the plurality of streams may bereceived from different communication client instances, each of thedifferent communication client instances being executed at a differentuser device. Each of the different communication client instances may bereceived from a different communication client instance executed on adifferent user device.

The computer may be embodied in a server.

The computer may be embodied in the user device.

According to a second aspect, the subject-matter of the presentapplication provides a computer for effecting a communication eventbetween a first user and one or more second users via a communicationnetwork, the computer comprising: a network interface configured toreceive, via the network, a plurality of video streams, each carrying arespective moving image of one or more users; a processor configured toperform operations of: causing the respective moving image of a first ofthe video streams to be displayed at a user device of the first user fora first time interval; detecting a change in the visual content of therespective moving image of a second of the video streams that is notbeing displayed at the user device during the first time interval; andin response to detecting said change in the visual content, causing therespective moving image of at least the second stream to be displayed atthe user device for a second time interval.

The processor may be configured to determine the duration of the secondtime interval based on the change in the visual content detected in themoving image of the second video stream.

The change may be detected by applying a facial detection algorithm tothe respective moving image of the second video stream to detect achange in the total number of faces in the moving image of the secondvideo stream during the first time interval.

Additionally or alternatively, the change may be detected by applying askeletal tracking algorithm to the respective moving image of the secondvideo stream to detect a change in the total number of skeletons in themoving image of the second video stream during the first time interval.

According to a third aspect, the subject-matter of the presentapplication provides computer program product for effecting acommunication event between a first user and one or more second usersvia a communication network, the computer program product comprisingcode stored on a computer readable storage medium and configured whenexecuted on a computer to perform the following operations: receiving,via the network, a plurality of video streams, each carrying arespective moving image of one or more users; causing the respectivemoving image of a first of the video streams to be displayed at a userdevice of the first user for a first time interval; detecting a changein the visual content of the respective moving image of a second of thevideo streams that is not being displayed at the user device during thefirst time interval; and in response to detecting said change in thevisual content, causing the respective moving image of at least thesecond stream to be displayed at the user device for a second timeinterval.

According to a fourth aspect, the subject matter of present applicationprovides a computer-implemented method of effecting a communicationevent between a first user and one or more second users via acommunication network, the method comprising implementing on a computerconnected to the network: receiving, via the network, a plurality ofvideo streams, each carrying a respective moving image of at least onerespective user; causing the respective moving image of a first of thevideo streams to be displayed at a user device of the first user for afirst time interval; identifying in the respective moving image of asecond of the video streams that is not displayed at the user device inthe first time interval, a human feature of the respective user;detecting a movement of the identified human feature during the firsttime interval that matches one of a plurality of expected movements; andin response to the detected movement, causing the respective movingimage of at least the second video stream to be displayed at the userdevice for a second time interval.

The computer may determine the duration of the second time intervalbased on which of the plurality of expected movements the movement ofthe identified human feature is detected as matching.

Each of the plurality of expected movements may be associated with apriority value, and the computer may use the priority value to selectthe second stream from the plurality of video streams for saiddisplaying at the user device for the second time interval.

The computer may be embodied in a server.

The computer may be embodied in the user device.

In some embodiments, causing at least the second video stream to bedisplayed at the user device may comprise replacing the first videostream with the second video stream, such that the first video stream isnot displayed at the user device for the second interval.

In other embodiments, both the first and second video streams may bedisplayed at the user device for the second interval.

The computer may be separate from the user device and the computer maycause the moving image of each of the first and second video streams tobe displayed at the user device, by transmitting that stream to the userdevice via the network for displaying thereat.

In further embodiments, a third video stream may be displayed at theuser device in the first time interval in addition to the first videostream and the third video stream may be replaced with the second videostream for the second interval, such that the third video stream is notdisplayed at the user device for the second interval.

The computer implemented method of the fourth aspect may also include:in response to detecting said movement, selecting a first of a pluralityof predetermined layouts for displaying at least the second video streamat the user device for the second time interval, wherein each of theplurality of predetermined layouts is for displaying a different numberof video streams at the user device, wherein a different one of thepredetermined layouts is used to display the first stream in the firsttime interval.

In some embodiments, the computer implemented method may cause audiodata associated with the first video stream to be played out at the userdevice during both the first and the second time intervals. The audiodata may be played out in the first and second time intervals inresponse to the computer detecting that the user in the moving image ofthe first video stream is speaking.

In further embodiments, at least two of the plurality of streams may bereceived from different communication client instances, each of thedifferent communication client instances being executed at a differentuser device. Each of the video streams may be received from a differentcommunication client instance executed on a different user device.

According to a fifth aspect, the subject-matter of the presentapplication provides computer for effecting a communication eventbetween a first user and one or more second users via a communicationnetwork, the computer comprising: a network interface configured toreceive, via the network, a plurality of video streams, each carrying arespective moving image of one or more users; a processor configured toperform operations of: causing the respective moving image of a first ofthe video streams to be displayed at a user device of the first user fora first time interval; identifying in the respective moving image of asecond of the video streams that is not displayed at the user device inthe first time interval, a human feature of the respective user;detecting a movement of the identified human feature during the firsttime interval that matches one of a plurality of expected movements; andin response to the detected movement, causing the respective movingimage of at least the second video stream to be displayed at the userdevice for a second time interval.

The computer may determine the duration of the second time intervalbased on which of the plurality of expected movements the movement ofthe identified human feature is detected as matching.

Each of the plurality of expected movements is associated with apriority value, and the computer uses the priority value to select thesecond stream from the plurality of video streams for said displaying atthe user device for the second time interval.

At least one of the plurality of expected movements may include amovement that corresponds to a user in the moving image of the secondvideo image; smiling, frowning, laughing, gasping, nodding their head,shaking their head, pointing in a particular direction with one or bothof their hands, waving with one or both of their hands, raising orlowering one or both of their arms above or below a predeterminedheight, clapping, moving one or more clenched fists so as to indicatecelebration or frustration, and giving a thumbs up or down with one orboth of their hands.

The computer of the fifth aspect may also include a processor configuredto perform the operation of: in response to detecting said movement,selecting a first of a plurality of predetermined layouts for displayingat least the second video stream at the user device for the second timeinterval, wherein each of the plurality of predetermined layouts is fordisplaying a different number of video streams at the user device,wherein a different one of the predetermined layouts is used to displaythe first stream in the first time interval.

According to a sixth aspect, the subject-matter of the presentapplication provides a computer program product for effecting acommunication event between a first user and one or more second usersvia a communication network, the computer program product comprisingcode stored on a computer readable storage medium and configured whenexecuted on a computer to perform the following operations: receiving,via the network, a plurality of video streams, each carrying arespective moving image of one or more users; causing the respectivemoving image of a first of the video streams to be displayed at a userdevice of the first user for a first time interval; identifying in therespective moving image of a second of the video streams that is notdisplayed at the user device in the first time interval, a human featureof the respective user; detecting a movement of the identified humanfeature during the first time interval that matches one of a pluralityof expected movements; and in response to the detected movement, causingat least the second video stream to be displayed at the user device fora second time interval.

CONCLUSION

Although the subject matter has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the subject matter defined in the appended claims is notnecessarily limited to the specific features or acts described above.Rather, the specific features and acts described above are disclosed asexample forms of implementing the claims.

1.-20. (canceled)
 21. A computer-implemented method comprising:receiving a plurality of data streams comprising data encoding videoimages; causing a first of the data streams to be rendered at a devicefor a first time interval; detecting a change in visual content of asecond of the data streams that is not being rendered at the deviceduring the first time interval, the change in the visual contentcomprising an entry or departure of at least one person; and in responseto detecting the change in the visual content, causing the second datastream to be rendered at the device for a second time interval.
 22. Thecomputer-implemented method of claim 21, further comprising determininga duration of the second time interval based on the change in the visualcontent detected in the second data stream.
 23. The computer-implementedmethod of claim 21, wherein the change in the visual content is detectedby applying a facial detection algorithm to second data stream to detecta change in the visual content of the second data stream during thefirst time interval.
 24. The computer-implemented method of claim 21,wherein the change in the visual content is detected by applying askeletal tracking algorithm to the second data stream to detect a changein the visual content of the second data stream during the first timeinterval.
 25. The computer-implemented method of claim 21, furthercomprising: assigning a first priority value to the second data streamfor the first time interval; in response to detecting the change in thevisual content, assigning a second priority value denoting a higherpriority to the second data stream; and selecting the second data streamfrom the plurality of data streams for rendering at the device based onthe second priority value.
 26. The computer-implemented method of claim21, wherein causing the second data stream to be rendered at the devicecomprises causing a respective moving image of the first data stream tobe replaced with a respective moving image of the second data stream.27. The computer-implemented method of claim 26, wherein causing thesecond data stream to be displayed at the device comprises causing therendering of the first data stream to be stopped at the device for thesecond time interval.
 28. The computer-implemented method of claim 21,wherein the first and second data streams are rendered simultaneously atthe device for the second time interval.
 29. The computer-implementedmethod of claim 21, wherein a third of the plurality of data streams isrendered at the device in the first time interval in addition to thefirst data stream, and wherein the method further comprises causing thethird data stream to be replaced with the second data stream such thatthe third data stream is not rendered at the device for the second timeinterval.
 30. The computer-implemented method of claim 21, furthercomprising in response to detecting the change in the visual content,selecting one of a plurality of predetermined layouts for rendering thesecond data stream at the device for the second time interval.
 31. Thecomputer-implemented method of claim 30, wherein the predeterminedlayouts are configured for rendering a number of data streams at thedevice.
 32. The computer-implemented method of claim 30, wherein adifferent one of the predetermined layouts is used to render the firstdata stream in the first time interval.
 33. The computer-implementedmethod of claim 21, wherein at least two of the plurality of datastreams are received from different communication client instancesexecuted at different devices.
 34. One or more computer-readable storagedevices comprising instructions stored thereon that, responsive toexecution by one or more processors, perform operations comprising:receiving a plurality of data streams; causing a first of the datastreams to be displayed at a device for a first time interval; detectinga change in visual content of second of the data streams that is notbeing displayed at the device during the first time interval, the changein the visual content comprising a change in a total number of persons;and in response to detecting the change in the visual content, causingthe second data stream to be displayed at the device for a second timeinterval.
 35. The one or more computer-readable storage devices of claim34, further comprising instructions that, responsive to execution by theone or more processors, perform operations comprising determining aduration of the second time interval based on the change in the visualcontent detected in the second data stream.
 36. The one or morecomputer-readable storage devices of claim 34, wherein the change in thevisual content is detected by applying a facial detection algorithm tothe second data stream to detect a change in a total number of facesduring the first time interval.
 37. The one or more computer-readablestorage devices of claim 34, wherein the change in the visual content isdetected by applying a skeletal tracking algorithm to the second datastream to detect a change in a total number of skeletons during thefirst time interval.
 38. A system comprising: one or more processors;and one or more computer-readable storage devices having instructionsstored thereon that, responsive to execution by the one or moreprocessors, cause the system to perform operations including: receivinga plurality of data streams comprising data encoding video images;causing a first of the data streams to be rendered at a device for afirst time interval; detecting a change in visual content of a second ofthe data streams that is not being rendered at the device during thefirst time interval, the change in the visual content comprising anentry or departure of at least one person; and in response to detectingthe change in the visual content, causing the second data stream to berendered at the device for a second time interval.
 39. The system ofclaim 38, wherein the operations further include determining a durationof the second time interval based on the change in the visual contentdetected in the second data stream.
 40. The system of claim 38, whereinthe change in the visual content is detected by applying a facialdetection algorithm to the second data stream to detect a change in atotal number of faces during the first time interval.